1. Field of the Invention
The present invention relates to an image processing method, and more particularly to an image processing method in which defects in the color tone of the pupils of a photographed object in a digital image are detected and corrected.
2. Description of the Related Art
Hitherto, there have may many cases in which objects have been photographed such that the color of the photographed object in an image is different from the color of the actual object due to the state of reflection of light during the photographing operation. The foregoing phenomenon, called xe2x80x9cred-eyexe2x80x9d, looks very unnatural and deteriorates the appearance of the image. Therefore, an image processing is performed to correct the images so that it looks more natural.
If a person is photographed from the front by using an electronic flash, the so-called xe2x80x9cred-eyexe2x80x9d phenomenon occurs which causes the pupils to be photographed red or gold. The red-eye phenomenon occurs when light of the electronic flash is made incident from the front on the opened eyes of a person in a dark place. In the foregoing state, light is regularly reflected by the eyes. The foregoing state appears in the photographed image. The red-eye phenomenon includes xe2x80x9cred-eyexe2x80x9d in which the pupils are photographed red and xe2x80x9cgold-eyexe2x80x9d in which the pupils are photographed gold. (Hereinafter, these two phenomena are collectively called xe2x80x9cred-eyexe2x80x9d.)
Since red-eye deteriorates the quality of the photograph, a variety of image processing methods for correcting red-eye have been disclosed. For example, a method has been disclosed in Japanese Patent Application Laid-Open (JP-A) No. 7-72537 in which regions including the eyes, which are regions at which red-eye is to be corrected, are designated. Then, threshold value processing of the saturation, luminance and the hue in the designated regions is performed. If pixels are in the regions predetermined threshold values, a determination is made that there is red-eye, and the red-eye is corrected. Another red-eye correction method has been disclosed in Japanese Patent Application Laid-Open (JP-A) No. 9-261580 in which pupil candidate regions are selected in accordance with color information and saturation information about the inside portions of regions surrounded by edges. Then, all of the pupils having defective color tone among the selected pupil candidate regions are corrected so that red-eye is corrected.
The conventional methods disclosed in Japanese Patent Application Laid-Open (JP-A) No. 7-72537 and Japanese Patent Application Laid-Open (JP-A) No. 9-261580 are arranged to perform threshold value processing of the saturation, luminance and the hue so as to determine the red-eye regions which must be corrected. Since the saturation, luminance and the hue of red-eye vary widely, errors in extraction and omissions from extraction cannot be prevented. There is also the concern that a strange image may be formed because flesh color portions may also be blackened in as a result of the process for correcting the red-eye of the pupils.
When extraction of edges of an image is performed as in Japanese Patent Application Laid-Open (JP-A) No. 9-261580, there is the concern that excess divisions or errors in dividing the regions may occur because an image has considerably complicated edges generally.
That is, sectioning into regions is not performed in Japanese Patent Application Laid-Open (JP-A) No. 7-72537 former method. Whether or not red-eye has occurred is determined in accordance with only the threshold values of colors. Therefore, it is easy for a region which does not require correction to be erroneously detected as a region which must be corrected together with a region which must be corrected. Further, Japanese Patent Application Laid-Open (JP-A) No. 9-261580 cannot accurately section the regions. Therefore, there arises the problem that it is difficult to correct only the pupil region which is the area which is the object of correction.
Hitherto, printing onto a photosensitive material of an image photographed on a photographic film (hereinafter called a xe2x80x9cfilmxe2x80x9d), such as a negative film or a reversal film, has been performed by direct exposure in which images on the film are projected to planarly expose of the surface of the photosensitive material. On the other hand, a digital photographic system has been put into practical use in which image information photographed on, for example, a film is photoelectrically read are converted into digital data which is then subjected to a variety of image processings. Thereafter, the photosensitive material is exposed in a digital manner.
The foregoing digital photographic system basically includes an image reading section, such as a scanner, for reading an image recorded on an original, such as a translucent original or a reflected original, to produce corresponding image data; an image processing section for subjecting image data inputted thereto to a variety of image processings and an image recording section having a printer for digitally exposing the photosensitive material in accordance with the to processed image data and a processor for developing the exposed photosensitive material and the like.
In the digital photographic system, the image of an original is read by the scanner. The read image is converted into digital image data. In the image processing section, image processing conditions (exposing conditions) are determined in accordance with the image data, and then a variety of image processings are performed. In the printer, the photosensitive material is scanned and exposed in accordance with processed image data so that a latent image is recorded. Then, a development process suitable for the photosensitive material is performed by the processor. As a result, a finished print is produced in which the image photographed on the film is reproduced.
The digital photographic system is able to shorten the time required to complete the exposure, that is, can quickly complete the exposure. Moreover, image information of the finished print can be stored on a recording medium, such as a magneto-optical disk. Therefore, re-reading of the film is not required and the operation for printing extra copies can quickly and easily be performed. Moreover, image processings such as editing operations (e.g., image synthesis and image division) and color/density adjustment can arbitrarily be performed. Therefore, an advantage can be realized in that a finished print which has been freely subjected to editing and image processings as needed can be output.
One of the important factors influencing the quality of the image of a portrait photograph is the reproducibility of the eyes of a human being. In particular, the red-eye phenomenon in which the eyes are photographed red due to the effects of the electronic flash or the like is a serious problem. Even in a case of a photograph of a satisfactory quality and free from the red-eye phenomenon, light must be reflected and shone in each eye. That is, if the catch light in the eye is expressed in the photograph too weakly, there arises a problem in that a portrait photograph having a satisfactory image quality cannot be obtained.
To solve the above-mentioned problems, the applicant of the present application proposed an image processing method and apparatus in Japanese Patent Application Laid-Open (JP-A) No. 10-75374. That is, the eye region is extracted from a region designated by an operator and including the eyes, and then determination of red-eye of the eye region is performed. If it is determined that there is red-eye, a red-eye eliminating process is performed. A catch light determination is performed after the eye region has been extracted. If a determination is made that the catch light is too weak, a process for enhancing the catch light is performed.
The above-described image processing method and apparatus disclosed in Japanese Patent Application Laid-Open (JP-A) No. 10-75374 enable correction of dark eyes to be performed by lowering the saturation of the eye region to make the color close to an achromatic color in a case of dark eyes. However, in a case of blue eyes, for example, the desired hue of blue is instructed in advance and stored, and conversion of the hue is carried out so as to perform correction for blue eyes. However, there arises a problem in that the difference in the hue from that of the eye of the photographed person can be recognized in the case of a close-up photograph of the eyes. In this case, an unsettling image is formed.
Accordingly, a first object of the present invention is to provide an image processing method which is able to accurately divide regions of any image to select a region which must be corrected as an object-of-concern region(a region to be corrected) A second object of the present invention is to provide an image processing method which is capable of accurately selecting a pupil region. A third object of the present invention is to provide an image processing method which is capable of accurately correcting a pupil region which is an object-of-correction region. A fourth object of the present invention is to provide an image processing method which is capable of producing a finished print with a natural feel from a corrected image. A fifth object of the present invention is to provide an image processing method which is capable of overcoming the problems experienced with conventional techniques and naturally correcting a portion to be corrected of an image, for example, red-eye, without resulting in a print with an unnatural feeling.
To achieve the first object, an image processing method of the present invention comprises the steps of: making an image region, which includes an eye region having discoloration and which has been designated in advance, to be an xy plane, and obtaining an image characteristic amount for each pixel by using anyone of hue, saturation and lightness or a combination of two or more of the hue, the saturation and the lightness; setting a three-dimensional xyz space on whose z axis are plotted the image characteristic amounts, and sectioning the xy plane of each region whose z axis values are distributed in a crest-shape with respect to the spread of the xy plane; and determining a discoloration region of a pupil in accordance with any one of or a combination of two or more of information about the shape, information about the position, information about the surface area and a statistical image characteristic amount on the xy plane of each of the sectional regions, and correcting the region determined to be a discoloration region to an image of a normal eye.
That is, the method according to the present invention is a method of collectively extracting and correcting discoloration regions, such as red-eye regions including a catch light portion. The closer to the central portion of the pupil, the greater the reflection of light off of the retina. Thus, in red-eye portions the lightness tends to decrease from the central portion to the periphery. Therefore, a crest-shape distribution of the lightness including the catch light is used. Moreover, the fact that the iris portion of a brown-type pupil is the valley of the lightness distribution is used. On the other hand, the degree of red of a blue-eye-type pupil is the valley from the pupil portion of the red-eye.
That is, the characteristic amount obtained by combining the lightness and the degree of red is used. The method of the present invention also utilizes the fact that a valley of the characteristic amount is formed between the red-eye portion, the white portion adjacent the red-eye portion, and the skin portion. Each crest portion of the characteristic amount is sectioned into regions so that the red-eye portion is separated from the whites-of-the-eyes portions and the skin portion.
In a second aspect of the present invention, sectioning of the xy plane of the image region distributed in each crest-shape is performed by: carrying out a number assigning process for each of the pixels within the image region designated in advance such that pixels within the image region designated in advance are grouped into reference regions each having pixels of N rowsxc3x97M columns (where each of N and M is not smaller than one) created around a pixel of interest which is the pixel in the reference region to which a number is to be assigned, and if the characteristic amount of all of the pixels in the reference region, the pixels of interest is considered to be the peak of a crest and a new number is assigned thereto, whereas if the characteristic amount of the pixels of interest is not the greatest characteristic amount of all of the pixels in the reference region and the pixel other than the pixel of interest in the reference region having the greatest characteristic amount has an assigned number, the assigned number is assigned to the pixel of interest as well, and this number assigning process is repeated until all of the pixels in the image region designated in advance are each assigned a crest peak number, and setting pixels having the same assigned number as respective regions, and sectioning the image region designated in advance into these respective regions. Since this method enables the division into regions to be performed by a computer program by a computer program, there is no need for the user to perform the complicated dividing operation.
In a third aspect of the present invention, sectioning of the xy plane of each region having a crest-shape distribution is performed by repeating a process in which, among pixels in the image region designated in advance, apixel which has not been assigned a number is employed as a pixel of interest, and when a pixel having a characteristic amount which is larger than that of the pixel of interest is detected in a reference region of pixels of N rowsxc3x97M columns (where each of N and M is not smaller than one) centered around the pixel of interest, the position of the pixel of interest is stored, and the pixel having the large characteristic amount is employed as a new point of interest, a number assigning process is carried out in which, when the new point of interest has the largest characteristic amount in the reference region and the new point of interest has not been assigned a number, the new point of interest is employed as the peak of a crest and a new number is assigned to the point of interest, and when a number has already been assigned to the new point of interest, the number is assigned to all pixels having stored coordinates, and this number assigning process is repeated until all pixels in the image region designated in advance are each assigned a crest number, and the image region is sectioned by a set of pixels having the same number being made to be one region.
The coordinates of the position of a pixel to which a number cannot be given are stored. Then, the number of the pixel which is the final new point of interest is given to all of the pixels having the stored coordinates. Thus, the process for sectioning into regions each of the crests of the characteristic amount can quickly be completed.
In a fourth aspect of the present invention, the determination of a discoloration region of the pupil is performed by determining for each sectioned region at least any one of the following five marks: a first mark which serves as information about the shape of the sectioned region by using a characteristic amount of the degree of roundness, and the greater the degree of roundness, the higher the first mark, a second mark which serves as information about the position of the sectioned region, and the closer the center of gravity of the sectioned region is to the center of the designated region, the higher the second mark, a third mark which serves as information about the surface area of the sectioned region, and the more the ratio of the surface area of the sectioned region and the surface area of the designated region deviates from a predetermined range, the lower the third mark, a fourth mark which serves as a statistical image characteristic amount expressing a degree of poorness of the color tone of the sectioned region, the fourth mark being obtained by comparing at least one of an average value, a maximum value, a minimum value, contrast and histogram shape of at least one of the hue, saturation and lightness of the second region with statistical information on regions having discoloration, a fifth mark which serves as information about the position of the sectioned region and which is obtained in accordance with the previously designated position of the center of the pupil and the distance between the two eyes, and the further the position of the sectioned region from the center of the pupil, the lower the fifth mark, and the sectioned region having the highest mark is determined to be a discoloration region.
That is, according to the fourth aspect of the present invention, in the method of the first aspect in which a discoloration region such as a red-eye region is determined from sectioned regions, the statistical characteristic amounts of the shape, area, position and the density of each sectioned region are converted into the first to fifth marks which tend to increase the closer the sectioned region is to the pupil and decrease the further the sectioned region is from the pupil. At least one of the first to fifth marks is used to judge whether the sectioned region is a discoloration region.
The relationship between the distance between two eyes and the diameter of the pupil is such that if the distance between the eyes is multiplied by a predetermined coefficient (gradually 0.07 to 0.11), the resulting product is the diameter of the pupil. On the basis of this relationship, the diameter of the pupil is calculated from the center of the pupil designated in advance and the distance between the eyes. A circular region having this diameter is given the highest fifth mark. The fifth mark decreases the further the sectioned region is from this center.
It is preferable that the L (L is an integer not smaller than one) regions having the highest averages or weighted averages of two or more marks are determined to be discoloration regions, as in the fifth aspect of the present invention. Since the weighted average is calculated, the respective regions are differentiated from one another even more. Therefore, a discoloration region can accurately be determined.
To achieve the third and fourth objects, a sixth aspect of the present invention comprises the steps of: making an image region, which includes an eye region having discoloration and which has been designated in advance, to be an xy plane, and obtaining an image characteristic amount for each pixel by using any one of hue, saturation and lightness or a combination of two or more of the hue, the saturation and the lightness; setting a three-dimensional xyz space on whose z axis are plotted the image characteristic amounts, and sectioning the xy plane of each region whose z axis values are distributed in a crest-shape with respect to the spread of the xy plane; and determining a discoloration region of a pupil in accordance with any one of or a combination of two or more of information about the shape, information about the position, information about the surface area and a statistical image characteristic amount on the xy plane of each of the sectional regions, and performing correction including a process for having gradation such that the lightness and/or saturation is gradually lowered from the periphery of the determined discoloration region of the pupil to the central portion so as to correct the determined discoloration region of the pupil to an image of a normal eye.
That is, the sixth aspect of the present invention determines the discoloration region by a method similar to that according to the first aspect. Therefore, a discoloration region, such as a red-eye portion, can accurately be separated from the whites-of-the-eyes portions and the skin portion. Thus, correction can accurately be performed.
To achieve the fourth object, the sixth aspect of the present invention is structured to have gradation such that the lightness and/or saturation is gradually lowered from the periphery to the central portion. As a result, an atomosphere can be realized which is similar to the color of the actual pupil portion in which the color in the central portion is darker than the peripheral portion. Therefore, a corrected image of the pupil which has a natural look can be obtained. Since the seventh to the ninth aspects of the present invention have similar operations as those of the fifth aspect, description thereof is omitted.
A tenth aspect of the present invention comprises the steps of: making an image region, which includes an eye region having discoloration and which has been designated in advance, to be an xy plane, and obtaining an image characteristic amount for each pixel by using any one of hue, saturation and lightness or a combination of two or more of the hue, the saturation and the lightness; setting a three-dimensional xyz space on whose z axis are plotted the image characteristic amounts, and sectioning the xy plane of each region whose z axis values are distributed in a rest-shape with respect to the spread of the xy plane; and determining a discoloration region of a pupil in accordance with any one of or a combination of two or more of information about the shape, information about the position, information about the surface area and a statistical image characteristic amount on the xy plane of each of the sectional regions, and adjusting the position having the highest lightness in the determined discoloration region to be a catch light position, and performing correction including a process for forming a catch light pattern at the catch light position, so as to correct the determined discoloration region of the pupil to an image of a normal eye.
Similarly to the sixth aspect, the tenth aspect of the present invention determines a discoloration region by a method similar to that according to the first aspect. Therefore, a discoloration region, such as a red-eye portion, can accurately be separated from the whites-of-the-eyes portions and the skin portions. Thus, correction can accurately be performed.
According to the tenth aspect of the present invention, the position in the pupil portion having the highest lightness is determined to be the catch light position, and a catch light pattern is formed. Since the catch light portion is a region having a low density in the pupil portion which has a high density, the position of the catch light is the brightest position. When a process for forming the catch light at the position in the pupil region having the highest lightness is performed, a natural and lively image of the eye can be obtained.
Since the eleventh to fifteenth aspects of the present invention have similar operations as those of the second to fifth aspects, description thereof is omitted.
A sixteenth aspect of the present invention comprises the steps of: making an image region, which includes an eye region having discoloration and which has been designated in advance, to be an xy plane, and obtaining an image characteristic amount for each pixel by using any one of hue, saturation and lightness or a combination of two or more of the hue, the saturation and the lightness; setting a three-dimensional xyz space on whose z axis are plotted the image characteristic amounts, and sectioning the xy plane of each region whose z axis values are distributed in a crest-shape with respect to the spread of the xy plane; and determining a discoloration region of a pupil in accordance with any one of or a combination of two or more of information about the shape, information about the position, information about the surface area and a statistical image characteristic amount on the xy plane of each of the sectional regions and enlarging/reducing normal color tone pupils cut out from a normal pupil region to have a size which coincides with the size of the determined discoloration region, and performing correction including a process for pasting the normal color tone pupils to the pupil region determined to be a discoloration region, as to correct the discoloration region of the pupil to an image of a normal eye.
That is, in the sixteenth aspect of the present invention, a pupil having a normal color and cut from a normal pupil region is enlarged or reduced and pasted onto the pupil region determined to be a discoloration region. Therefore, a relatively simple correction process is able to correct a discoloration region of the pupil to a normal image of an eye. Since the seventeenth to twentieth aspects of the present invention have similar operations as those of the second to fifth aspects, description thereof is omitted.
To achieve the fourth object, in a twenty-first aspect, in the image processing method of any one of first to twentieth aspects of the present invention, further correction is performed such that the atmosphere of the image of the eye including a corrected pupil portion and the state of the image of portion of eye for which correction is unnecessary are the same, or such that when discoloration of both eyes has been corrected, the states of the both eyes are the same.
The position of an eye which does not require correction and the position of a corrected eye are compared with each other. Then, fine correction can be performed such that the position of the corrected image of the eye is shifted to a natural position or the color of the corrected eye is made to be the same as the color of the eye which does not require correction. Therefore, an image having a natural look can be formed. Moreover, the position of the catch light is corrected to make the positions of catch light in the pupils to be the same.
In a twenty-second aspect of the present invention, the red-eye correction is carried out plural times. Namely, in the twenty-second aspect, in the image processing method of any one of first to twenty-first aspects of the present invention, in accordance with number of times of a designated region including an eye region having discoloration, the characteristic amount for use in a method of sectioning an image or sectioning of a region is changed, or the type of the characteristic amount for use in determining a discoloration region, or a method of calculating the characteristic amount, or a criterion for judgment is changed, or a method of correcting a discoloration region is changed. For example, when the determination of the red-eye region is performed by dividing into regions each crest of the characteristic amount, the second determination of the red-eye region is performed on the basis of the similarity of the color tones, in stead of dividing each crest of the characteristic amount into regions. Alternatively, the degree of roundness, which is used in the first determination, is not employed or the surface, which is used to perform the first determination, is not employed.
To achieve the foregoing objects, according to a twenty-third aspect of the present invention, there is provided an image processing method applied to a digital photographic system for making a photographic print by subjecting digital image data to image processing, said image processing method comprising the steps of: selecting an image which is to be corrected, and an image including a characteristic portion corresponding to a portion to be corrected of the image to be corrected; designating a region including the portion to be corrected of the image to be corrected, and a region including the characteristic portion of the selected image; and using the characteristic portion of the selected image to correct the portion to be corrected of the image to be corrected.
When the image to be corrected is corrected, it is preferable that the image to be corrected, the selected image and a corrected image are and displayed together on an image display portion of the digital photographic system.
It is preferable that correction of the image to be corrected includes correction of at least one of red-eye, wrinkles, blotches, freckles, pimples, scars, and burn marks.